Vehicle path restoration system through sequential image analysis and vehicle path restoration method using the same

ABSTRACT

Disclosed are a vehicle path restoration system through sequential image analysis which includes: an image capturing unit that acquires sequential images from the front camera installed in the subject vehicle; an image analysis unit for generating multiple lanes that can be recognized from the sequential images of the video file acquired by the image capturing unit and multi-paths calculated using the geometric characteristics of the lanes recognized at the current time and the speed of the subject vehicle, that restores the path of the subject vehicle and restores the path of the front vehicle driving in front of the subject vehicle; a memory for storing path data of the subject vehicle and the front vehicle restored by the image analysis unit; and a display unit that expresses the path data of the subject vehicle and the front vehicle stored in the memory in the form of a top view.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Korean Patent Application No.10-2022-0005776, filed on Jan. 14, 2022, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference in its entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a vehicle path restoration systemthrough sequential image analysis and a vehicle path restoration methodusing the same, and more specifically, a vehicle path restoration systemand a vehicle path restoration method using the same which is expressedin a top view by restoring surrounding situations such as the moment ofa traffic accident through sequential image (video) analysis.

Background Technology Description of the Related Art

In the case of a traffic accident, it is often very ambiguous todistinguish between the offender and the victim. Therefore, there is anongoing dispute over who is the offender and who is the victim.

Recently, with the increase in the supply of automobiles and theincrease in casualties due to accidents, black boxes that have been usedfor aviation are also being used in vehicles. A vehicle black boxenables the cause of a traffic accident to be clearly determined byanalyzing related data such as vehicle speed, direction, and brakeoperation.

Such a vehicle black box records the situation at the time of theaccident with a camera installed on the front or rear of the vehicle,and stores all sounds around it with a microphone.

However, there is a problem in that it is difficult to accuratelyunderstand the path between the actual driving vehicle and the vehiclein front at a glance. That is, there is a need for a means to accuratelydetermine the cause of the accident by restoring the entire path.

CITED REFERENCE Patent Document

-   (Patent Document 1) Korean Patent No. 10-1342124 (A Front Vehicle    Detecting And Tracking System Using The Image And Detecting And    Tracking Method Using The Same, Dec. 19, 2013)-   (Patent Document 2) Korean Patent No. 10-1455835 (Lane Recognition    and Tracking System Using Images, And Method For Recognition And    Tracking Lane Using The Same, Nov. 4, 2014)-   (Patent Document 3) Korean Patent No. 10-1473866 (IMAGE PROCESSING    SYSTEM FOR VEHICLE AND IMAGE PROCESSING METHOD OF USING THE SAME,    Dec. 17, 2014)-   (Patent Document 4) Korean Patent No. 10-2296520 (METHOD OF    DETECTING CURVED LANE THROUGH PATH ESTIMATION BY MONOCULAR VISION    BASED CAMERA, Sep. 1, 2021)-   (Patent Document 5) Korean Patent No. 10-2318586 (METHOD OF    DETECTING MEDIAN STRIP AND PREDICTING COLLISION RISK THROUGH    ANALYSIS OF IMAGES, Oct. 28, 2021)

SUMMARY OF THE INVENTION

The present invention has been devised in view of the above-describedproblems, and its purpose is to restore the surrounding conditions suchas the moment of a traffic accident through sequential image analysisand express it in a top view. It is to provide a vehicle pathrestoration system through sequential image analysis and a vehicle pathrestoration method using the same.

The vehicle path restoration system through sequential image analysis ofthe present invention for solving the above problems includes: an imagecapturing unit that acquires sequential images from the front camerainstalled in the subject vehicle; an image analysis unit for generatingmultiple lanes that can be recognized from the sequential images of thevideo file acquired by the image capturing unit and multi-pathscalculated using the geometric characteristics of the lanes recognizedat the current time and the speed of the subject vehicle, that restoresthe path of the subject vehicle and restores the path of the frontvehicle driving in front of the subject vehicle; a memory for storingpath data of the subject vehicle and the front vehicle restored by theimage analysis unit; and a display unit that expresses the path data ofthe subject vehicle and the front vehicle stored in the memory in theform of a top view.

Preferably, the multi paths are located by a plurality of indexes in apredefined memory space, so that it is possible to store path data evenwhen the subject vehicle changes lanes left and right between the multipaths.

The vehicle path restoration method through sequential image analysis ofthe present invention for solving the above other problems includes:acquiring sequential images from a front camera of an image capturingunit installed in a subject vehicle; generating a path of a subjectvehicle and a front vehicle by receiving the sequential image andperforming image analysis in an image analysis unit; and expressing theimage analysis of the image analysis unit in the form of a top view onthe display unit; performing image analysis in the image analysis unitare: determining a lane of the current time; generating a multi-lanerecognizable in the front camera image and a multi-path for each indexcalculated using geometric characteristics of the lane recognized at thecurrent time and the speed of the subject vehicle; generating path dataof the subject vehicle; and generating path data of the front vehicle.

Preferably, a path set of each index in the multi-path comprisesreceiving input information that is input data for creating a path set;determining whether the path is a curved line or a straight lane;generating a curved path when the path is determined as a curved lane,and generating a straight path when the path is determined as a straightlane; and creating a path set of the corresponding index by integratingthe curved path and the straight path. The input data for creating thepath set are the speed of the subject vehicle, the path index number,the equation coefficient of a straight line and the equation coefficientof the curve suitable for the lane recognized in the image, the radiusof curvature, and the curve discrimination coefficient.

Preferably, the step of creating the multi-path for each index includes:determining an index of a path connected to the lane recognized in thecurrent frame in the front camera image as an active path index, anddetermining an index of a path not connected to the lane recognized inthe current frame in the front camera image as an inactive path index;the active path index is passed to the active multi-path to generate anactive multi-path; and the inactive path index is transferred to theinactive multi-path, and in the inactive multi-path generation, a newpath is not added, but according to the movement of the subject vehicle,correcting the location of the path stored in the past time. The activemulti-path generation is the entire process of a method of creating apath set of each individual index, and the active single path generationstep is performed several times as many as the number of lanesrecognized in the current image.

Preferably, generating path data of the subject vehicle comprises:obtaining equation coefficients of straight lines and the equationcoefficients of curve of a driving lane, which are both lanes of asubject vehicle; determining the equation coefficients of the straightline and the equation coefficients of curve for the path of the subjectvehicle by using the equation coefficients of the straight line and theequation coefficients of the curve of the driving lane; determiningwhether the path is a curved line or a straight lane; generating acurved path when the path is determined as a curved lane, and generatinga straight path when the path is determined as a straight lane; andgenerating a set of paths for the subject vehicle by integrating thecurved path and the straight path.

Here, further comprising the step of making the driving lane in aparallel state: determining a lane probability variable that is aprobability of being a lane with respect to a right lane and a leftlane; when the difference between the left lane and the right laneprobability variable is less than or equal to a predetermined errortolerance, obtaining a median value of the slope and curvature of thetwo lane information, and calculating the equations of straight linesand curves for the path with respect to the left and right lanes;comparing the sizes of the left lane and right lane probabilityvariables when the difference between the left and right laneprobability variables is out of a predetermined error tolerance; and ifthe right-lane probability variable is larger than the left-laneprobability variable, calculate the equations of straight lines andcurves for the left-lane path, such as the slope and curvature of theright-lane, Ii the right-lane probability variable is smaller than theleft-lane probability variable, calculate the equations of straightlines and curves for the right-lane path, such as the slope andcurvature of the left-lane.

Preferably, when the location of the multi-lane is changed because thesubject vehicle is changed to a lane, the information of the multi-laneis moved together.

Preferably, generating of the path data of the front vehicle includes:determining a vector b representing that the front vehicle moves from aspecific position of an image of a previous frame to a specific positionof an image of a current frame when the camera coordinates of thesubject vehicle are taken as the origin; determining a motion vector hof the subject vehicle connecting the current coordinates and thecoordinates of the previous frame in the path of the subject vehicle;and obtaining a motion vector v of the front vehicle that is a vectoraddition of the vector b and the vector h.

According to the present invention having the configuration as describedabove, it is possible to express a top view by restoring surroundingsituations such as the moment of a traffic accident through sequentialimage analysis.

In addition, it has various applications, such as determining thedriver's driving habits and whether the front vehicle violates trafficlaws by restoring the path between the subject vehicle and the frontvehicle.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The above object and advantages of the present invention will becomemore apparent by describing in detail exemplary embodiments thereof withreference to the attached drawings in which:

FIG. 1 is a diagram illustrating a vehicle path restoration systemthrough sequential image analysis according to an embodiment of thepresent invention.

FIG. 2 is a flowchart illustrating a vehicle path restoration methodthrough sequential image analysis according to an embodiment of thepresent invention.

FIG. 3 is a view illustrating recognition of multiple lanes and a frontvehicle in an input image according to an embodiment of the presentinvention.

FIG. 4 is a diagram illustrating a top view made from an input imageaccording to an embodiment of the present invention.

FIG. 5 is a diagram illustrating a multi-lane and multi-path accordingto an embodiment of the present invention.

FIG. 6 is a diagram illustrating a case in which paths are expanded to12 paths according to an embodiment of the present invention.

FIG. 7 is a view after changing one lane to the left in FIG. 6 accordingto an embodiment of the present invention.

FIG. 8 is a flowchart illustrating a method of creating one path set inthe image analysis unit according to an embodiment of the presentinvention.

FIG. 9 is a diagram illustrating a method of generating a curved pathaccording to an embodiment of the present invention.

FIG. 10 is a diagram illustrating a method of generating a straight linepath according to an embodiment of the present invention.

FIG. 11 is a diagram illustrating a process of forming a multi-path inan image analysis unit according to an embodiment of the presentinvention.

FIG. 12 is a diagram illustrating a lane probability variable accordingto an embodiment of the present invention.

FIG. 13 is a diagram illustrating an algorithm for making driving lanesparallel to each other using probability variables (P_(L), P_(R))according to an embodiment of the present invention.

FIGS. 14 to 17 are views continuously illustrating a state in which asubject vehicle changes a location from an initial lane to a left laneaccording to an embodiment of the present invention.

FIGS. 18 and 19 are diagrams illustrating a method of moving multi-laneinformation when a subject vehicle changes lanes.

FIG. 20 is a diagram illustrating a path of a subject vehicle accordingto an embodiment of the present invention.

FIGS. 21 to 23 are diagrams for explaining a method for generating apath of a front vehicle using a motion vector of a subject vehicleaccording to an embodiment of the present invention.

FIG. 24 is a view showing a top view by calculating a motion vector of afront vehicle according to an embodiment of the present invention.

FIG. 25 is a view illustrating a general state in which vectors of afront vehicle are connected to each other to form a path set accordingto an embodiment of the present invention.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

Hereinafter, vehicle path restoration system through sequential imageanalysis and vehicle path restoration method using the same according toa preferred embodiment of the present invention will be described indetail with reference to the accompanying drawings.

FIG. 1 is a diagram illustrating a vehicle path restoration systemthrough sequential image analysis according to an embodiment of thepresent invention.

Referring to FIG. 1 , sequential images are acquired by the imagecapturing unit 100 which is a front camera installed in a subjectvehicle.

The image analysis unit 200 analyzes the sequential images of the videofiles obtained by the image capturing unit 100 to generate a multi-lanepath and restore the path of the subject vehicle. In addition, the imageanalysis unit 200 restores the path of the front vehicle driving ahead.

The path data of the subject vehicle and the front vehicle is stored inthe memory 300. It is stored with respect to the time frame of thesequential image.

Finally, the above data is expressed in the form of a top view on thedisplay unit 400.

FIG. 2 is a flowchart illustrating a vehicle path restoration methodthrough sequential image analysis according to an embodiment of thepresent invention.

Referring to FIG. 2 , sequential images are acquired from the frontcamera of the image capturing unit 100 installed in the subject vehicle(S101).

Next, the image analysis unit 200 performs image analysis by inputtingsequential images (S102). The image analysis unit 200 includestechnologies such as lane recognition, vehicle recognition, and objectrecognition.

First, a lane of the current time for displaying the top view isdetermined (S1021).

A multi-lane path is generated for each index by using information onvarious objects on the road obtained through image analysis (S1022).

A path data of the subject vehicle is formed through image analysis(S1023).

A path data of the front vehicles are generated through image analysis(S1024).

Next, the information obtained in S1021-S1024 is expressed in the formof a top view (S103).

FIG. 3 is a view showing recognition of multiple lanes and a frontvehicle in the input image according to an embodiment of the presentinvention, and FIG. 4 is a view showing a top view made from the inputimage according to an embodiment of the present invention.

Referring to FIG. 3 , it shows recognition of multiple lanes and a frontvehicle in an input image. This operation is repeated for sequentialimages, and a path data set is generated from the geometricalcorrelation of the recognized lanes. Once this data set and the lanedata recognized in the current frame are combined and expressed in theworld coordinate system, a top view as shown in FIG. 4 can be created.

Referring to FIG. 4 , the four solid lines on the upper part mean thatfour multi-lanes are recognized as curves in the image (FIG. 3 ) of thecurrent frame, and the four dotted lines represent the four multi-pathsgenerated in previous time frames.

In addition, the vehicle picture located between the solid line and thedotted line shows the location of the subject vehicle 10 in which thecamera is installed. The vehicle picture displayed on the upper end ofthe subject vehicle 10 shows the position of the front vehicle 20(boundary box) detected in FIG. 3 .

The present invention analyzes the video in real time, and aims to showa lot of information at the moment of the accident. Therefore, the pastmulti-path 30 and the forward multi-lane 40 recognized at the presenttime are shown together.

(Multi-Lane and Multi-Path Structure)

Hereinafter, the structures of the multi-lane 40 and the multi-path 30will be described.

The multi-lane 40 means a lane recognizable in the front camera image,for example 4 lanes in FIGS. 3 and 4 .

FIG. 5 is a diagram illustrating a multi-lane and multi-path accordingto an embodiment of the present invention.

Referring to FIG. 5 , as a basic view of the multi-lane 40 and themulti-path 30, the picture of the vehicle in the center means thesubject vehicle 10 in which the camera is installed. The four straightlines in the upper part mean the multi-lane 40 recognized in the currentimage.

The circular number ({circle around (0)}{circle around (1)}{circlearound (2)}{circle around (3)}) shows that a number is assigned to eachmulti-lane 40.

The lanes {circle around (1)} and {circle around (2)} mean the left andright lanes recognized from the left and right sides of the subjectvehicle 10, and these are called driving lanes.

Lane {circle around (0)} and {circle around (3)} are lanes recognizedfrom the left and right sides of the driving lane and are calledextended lanes.

In addition, four dotted lines ([0][1][2][3]) existing under the subjectvehicle 10 mean the multi-path 30. Here, the past multi-path iscalculated using the geometrical characteristics of the lane recognizedat the present time and the speed of the subject vehicle.

For example, if the lane {circle around (1)} in the path [1] is astraight line, the predicted path can be calculated using the vehiclespeed and the linear function. If the lane {circle around (1)} in thepath [1] is a curve, the expected path can be calculated using thevehicle speed, the movement vector using the inscribed circle of theparabola, and the correction of the entire path etc. A detaileddescription of obtaining a movement path of a curved lane is disclosedin Korean Patent Registration No. 10-2296520 (corresponding U.S. patentapplication Ser. No. 17/159,150) of the present applicant.

FIG. 5 is an example illustrating a state in which a subject vehicletravels without changing a lane. In an actual driving environment, sincethe subject vehicle changes lanes, n lane paths may exist. In the actualinvention implementation process, the number of paths is limited due tolimitations in the amount of computation and memory.

FIG. 6 is a diagram illustrating a case in which paths are expanded to12 according to an embodiment of the present invention.

Referring to FIG. 6 , as a case in which 12 paths are expressed, the 12paths ([0] to [11]) are located in a memory space predefined forrecording the paths of lanes existing in the sequential image.

The orthogonal coordinate system displayed on the picture of the subjectvehicle in FIG. 6 means a coordinate system of a multi-lane andmulti-path and is a world coordinate system having the position of thecamera installed in the subject vehicle 10 as an origin. The subjectvehicle 10 may move left and right between these paths. In FIG. 6 , thesubject vehicle 10 currently located in the center can store path dataeven when changing lanes 5 times to the left and right, respectively.

FIG. 7 is a view after changing one lane to the left in FIG. 6 accordingto an embodiment of the present invention.

Referring to FIG. 7 , the multi-lane structure ({circle around(0)}{circle around (1)}{circle around (2)}{circle around (3)}) has movedone step to the left from the location of the previous path([4][5][6][7]) to ([3][4][5][6]).

In the present invention, a path set is formed for each multi-path index[0] to [11].

FIG. 8 is a flowchart illustrating a method of creating one path set inthe image analysis unit 200 according to an embodiment of the presentinvention.

Referring to FIG. 8 , the image analysis unit 200 receives inputinformation (S801).

The input information is input data for creating a path set.Specifically, the input data includes the speed of the subject vehicle,the path index number ([0] to [11] in FIG. 6 ), the equationcoefficients of a straight line (a, b) and the equation coefficients ofthe curve (c, d, e) suitable for the lane recognized in the image,radius of curvature, curve discriminant coefficient, etc. The curvediscriminant coefficient is a variable that is set to 0 if the lanerecognized in the image is a straight line, and set to 1 if it is acurved line.

Next, it is determined whether it is a curved or straight lane (S802).It is a conditional statement that divides into the next step accordingto the value of the curve discriminant coefficient.

The method to obtain the curve discriminant coefficient is as follows.For the skeleton coordinates constituting the lane, fit the equation ofa straight line or curve using the least squares method. In this case,the shortest distance d_(S) between the coordinates of the skeleton andthe equation of the straight line can be obtained. This shortestdistance can be obtained for the number N of all skeleton coordinates,and the average value can be obtained.

By applying the above contents, it can be expressed as in Equation 1,μ_(S) and can be called an error rate of a straight line with respect toa lane.

$\begin{matrix}{\mu_{S} = {\frac{1}{N}{\sum d_{S}}}} & \left\lbrack {{Equation}1} \right\rbrack\end{matrix}$

Similarly to the above method, the shortest distance d_(c) between theskeleton coordinates and the equation of the curve is obtained. Thisshortest distance can be obtained for the number N of all skeletoncoordinates, and the average value can be obtained. By applying theabove contents, it can be expressed as in Equation 2, and μ_(c) can becalled the error rate of the curve with respect to the lane.

$\begin{matrix}{\mu_{C} = {\frac{1}{N}{\sum d_{C}}}} & \left\lbrack {{Equation}2} \right\rbrack\end{matrix}$

If the error rate of the straight line is smaller than the error rate ofthe curve, the curve discriminant coefficient is set to 0, and if theerror rate of the straight line is greater than the error rate of thecurve, the curve discriminant coefficient is set to 1.

Next, when it is determined as a curved lane, a curved path is generated(S803). A method of generating a curved path is disclosed in KoreanPatent Registration No. 10-2296520 (corresponding U.S. patentapplication Ser. No. 17/159,150) of the present applicant.

In addition, when it is determined as a straight lane, a straight pathis generated (S804). A method of generating a straight path will bedescribed later.

Next, a path set of the corresponding index is generated by integratingthe above-described curved path and straight path (S805).

FIG. 9 is a diagram illustrating a method of generating a curved pathaccording to an embodiment of the present invention. Since the methodfor generating a curved path is disclosed in Korean Patent RegistrationNo. 10-2296520 (corresponding U.S. patent application Ser. No.17/159,150) of the present applicant, it will be briefly described.

Referring to FIG. 1 , a method of detecting a curved lane includesacquiring forward images of a curved lane by using a monocular visioncamera which captures the front from a vehicle, in operation S110,detecting curved lane candidates from the forward images, in operationS120, detecting a skeleton line having a thickness of 1 pixel by using amedian value of the curved lane candidates, in operation S130,estimating a parabolic function which is curve-fitted by using theskeleton line, in operation S140, generating an inscribed circleinscribing in a parabola that corresponds to the parabolic function at alocation where a world coordinate value Y is 0, in operation S150,generating a set of moving paths after anticipated coordinates ofrepresentative points in a current frame are calculated from coordinatesof representative points which are tangent points of the parabola in aprevious frame and the inscribed circle on the circumference of theinscribed circle, in operation S160, determining whether the skeletonline satisfies a specific condition for curve fitting in a currentframe, in operation S170, when the skeleton line does not satisfy thespecific condition, determining the parabola that fits the skeleton lineas a curved lane, in operation S180, and when the skeleton linesatisfies the specific condition, estimating the parabola which iscurve-fitted to all skeleton line and set of the moving paths as acurved lane, in operation S190. Accordingly, the curved lane may beestimated by using the moving paths on the curved lane in the past.

FIG. 10 is a diagram illustrating a method of generating a straight linepath according to an embodiment of the present invention.

Referring to FIG. 10 , P₁=(X₀, Y₀) is a representative point of thecurrent time at Y₀=0 with respect to the equation of the straight lineHere, the representative point means coordinates that can represent themovement path of the lane on a straight line. P₂=(X_(E), Y_(E)) is theexpected coordinates of the next time.

L _(c)=√{square root over ((X ₀ −X _(E))²+(Y ₀ −Y _(E))²)}  [Equation 3]

X _(E) =a+bY _(E)  [Equation 4]

L_(S) in Equation 3 means the Euclidean length of the vector {rightarrow over (P₁P₀)}. Here, since the unknown is (X_(E), Y_(E)), bysubstituting Equation 4 into Equation 3, a quadratic equation (Equation5) can be obtained.

(b ²+1)Y _(E) ²−2{b(X ₀ −a)+Y ₀ }Y _(E)+(X ₀ −a)² +Y ₀ ² −L _(S)²=0  [Equation 5]

Among the two values of Y_(E) obtained in Equation 5, a negative numberis selected and inputted in Equation 4 to obtain X_(E). Through theabove method, a path of a straight lane can be obtained.

FIG. 11 is a diagram illustrating a process of forming a multi-path inthe image analysis unit 200 according to an embodiment of the presentinvention.

Referring to FIG. 11 , the index of the path connected to the lanerecognized in the current frame is determined as the active path index(S1101), and the non-activated path is determined as the inactive pathindex (S1105).

What is important in the process of creating a multi-path is whether ornot a lane is detected in the corresponding path. For example, in FIG. 6, there are 4 lanes detected in the image, which are connected to thepath [4][5][6][7]. If this state is maintained, in the path index[4][5][6][7], new representative points are added to the path everyframe, and a new path will be displayed on the screen of the top view.At this time, it is defined that the paths are activated.

The active path index is transferred to the active multi-path (S1102).

In the active multi-path, each active single path is generated (S1103,S1104). Here, each active single path generation (S1103, S1104) is theentire process of the method for creating the path set described in FIG.8 . That is, the active single path generation step is performed severaltimes as many as the number of lanes recognized in the current image.

When the index of the path connected to the lane recognized in thecurrent frame is determined as the inactive path index (S1105), it istransferred to the inactive multi-path (S1106). In the inactive singlepath generation (S1107, S1108), a new path is not added, but thelocation of the path stored in the past time is corrected according tothe movement of the subject vehicle.

The remaining paths [0][1][2][3][8][9][10][11] other than the lane path[4][5][6][7] detected in FIG. 6 do not connect with the current lanes,so no new paths are added to the top view screen. At this time, it isdefined that the paths are inactive. However, existing paths in theinactive paths are continuously modified according to the movement ofthe subject vehicle. Therefore, the step of generating the inactivemulti-path (S1106) in FIG. 11 is also executed for every frame.

(How to Restore Multi-Lane to Parallel State)

A method is required to ensure that multiple lanes remain parallel.

If the result of lane recognition is used as it is for path generation,multi-lane paths cannot be maintained parallel to each other. Forexample, the slopes of the left and right lanes obtained by imagerecognition in a straight section cannot completely match. (See FIG. 3 )When these data are used as path data as they are, the left and rightpaths may become closer or farther away from each other as time passes.

To solve the above problem, lane information for a path managedseparately from lane information recognized as an image is generated. Inthe multi-lane case, both lanes of the subject vehicle as a referenceare called driving lanes.

Although the average value of the information of the two driving lanesmay be used, in this case, incorrect lane information may be input andan error of the path may occur.

Hereinafter, a method for reducing a path error will be described.

In order to create a driving lane in a parallel state, a reference lanemust be selected among the two lanes. The reference lane may mean a lanehaving a higher probability of being the lane among the two lanes.

The probability that the data detected in the image is a lane is definedas a lane probability variable.

FIG. 12 is a diagram for explaining a lane probability variableaccording to an embodiment of the present invention.

Referring to FIG. 12 , a dotted line is geometrically expressed. Thelower lane is L₂, and the upper lane is L₂. And, an empty sectionbetween these two lanes is expressed by a dotted line.

The lane probability variable of the present invention will bedescribed.

The first probability is the ratio of the number of fitted skeleton lineto the total number of skeleton line. A method of making a skeleton lineis disclosed in Korean Patent Registration No. 10-2296520 (correspondingU.S. patent application Ser. No. 17/159,150) of the present applicant.

The first probability is about how long the lanes of the image look.This can be expressed as a mathematical expression as follows.

$\begin{matrix}{p_{1} = \frac{n_{s}}{N}} & \left\lbrack {{Equation}6} \right\rbrack\end{matrix}$

Here, N denotes the maximum length in which the lane skeleton line canexist in the image. n_(S) means the length of the skeleton line of thelane detected in the current image.

The second probability relates to how close the y₂ ^(d) value of thelower coordinate c₁ ^(d) of the skeleton line L₁ is to the subjectvehicle. This means that the closer the skeleton line is to the subjectvehicle, the higher the probability of obtaining the correct lane.

Equation 7 shows how to obtain the probability p₂.

$\begin{matrix}{d = {❘{y_{1}^{d} - y_{0}^{d}}❘}} & \left\lbrack {{Equation}7} \right\rbrack\end{matrix}$ $p_{2} = \frac{1}{{w \times d} + 1}$

Here, y₀ ^(d) means the Y-coordinate of the lowermost end of the image.

As the y₁ ^(d) value of the dotted line is closer to the bottom of theimage, the value of approaches 1, and as the y₁ ^(d) value of the dottedline moves away from the bottom of the image, the value of p₂ approaches0. w is a weight value for adjusting the probability value. For example,when y₁ ^(d) exists at about 30 m, if you want to make probability p₂0.5, you can set w to 1/30.

The lane probability variable P for one lane means the joint probabilityof the probabilities p₁ and p₂.

Since the probabilities p₁ and p₂ and are independent events, they canbe expressed by Equation 8.

P=p ₁ ×p ₂  [Equation 8]

Since the driving lane consists of left and right, the left lane randomvariable is and the right lane random variable is p_(R).

The relationship between the two lane probability variables (P_(L),P_(R)) can be defined in three ways.

The first relationship is when two lane probability variables havesimilar values. It can be expressed as a conditional expression as inEquation 9.

|P _(L) −P _(R)|<θ  [Equation 9]

In Equation 9, θ is the difference between the lane probabilityvariables of the left and right lanes, and means an error tolerancevalue. For example, if θ is 0.2, if the difference between the laneprobability variables of the left lane and the right lane has a value of0.2 or less, it may be determined that they are similar.

The second relation is the case where P_(R) is greater than P_(L), andthe third relation is the case where P_(R) is less than P_(L).

FIG. 13 is a diagram illustrating an algorithm for making driving lanesparallel to each other using lane probability variables (P_(L), P_(R))according to an embodiment of the present invention.

Referring to FIG. 13 , it is determined whether the difference betweenthe lane probability variables of the left lane and the right lane,which is the condition of Equation 9, is equal to or less than θ(S1301).

If Equation 9 is satisfied, a median value of the slope and curvature ofthe two lane information is obtained, and the equations of the straightline and the curve for the path are calculated for the left and rightlanes (S1302).

If Equation 9 is not satisfied, the values of P_(L), and P_(R) arecompared (S1303).

When P_(R), is greater than P_(L), the equation of straight line andcurve for the path of the left lane, such as the slope and curvature ofthe right lane, is calculated (S1304).

When P_(R) is less than P_(L), equation of straight line and curve forthe path of the right lane, such as the slope and curvature of the leftlane, is calculated (S1305).

The form of the equation of the straight line (Equation 10) and theequation of the curve (Equation 11) used in the present invention is asfollows.

x=a+by  [Equation 10]

x=c+dy+ey ²  [Equation 11]

Coefficients a, b, c, d, and e in Equations 10 and 11 denote the resultsof image recognition, where a and b are coefficients of a straight line,and c, d, and e are coefficients of a parabola.

a^(v), b^(v), c^(v), d^(v), e^(v), which will be described later, isdefined as an equation coefficient of a straight line and a curve for apath.

In the case of step S1302 in FIG. 13 , since two lanes have similarshapes, an equation of a straight line and a curve for a path iscalculated by calculating a median value of the reference lane among thetwo lanes.

In the case of a straight line, the intermediate value of the slope b ofthe straight line of the two lanes is set as the slope of the referencelane. This slope is set to b^(v), and the coefficient a^(v) for astraight line passing through the skeleton line coordinates of the laneis obtained using the least square method. At this time, the equation ofa straight line of the left and right lanes can be obtained, and this isused for path generation. The coefficient of the left path straight lineis a_(L) ^(v), and the right path straight coefficient is denoted asa_(R) ^(v).

In the case of the curve, the median value of the slope d of the curveof the two lanes is set as the slope d^(v) of the reference lane. Also,an intermediate value of the curvature e of the curves of the two lanesis set as the curvature e^(v) of the reference lane. The slope d^(v) andthe curvature e^(v) are fixed as constant values, and the curvecoefficient for a path passing through the skeleton coordinates of thelane is calculated using the least squares method. At this time, theequation of the curve of the left and right lanes can be obtained, andthis is used for path generation. The coefficient of the left path curveis c_(L) ^(v), and the coefficient of the right path curve is denoted asc_(R) ^(v).

Steps S1304 and S1305 in FIG. 13 are cases in which Equation 9 is notsatisfied, which means a case in which either lane is determined as thereference lane.

In the case of step S1304, the lane probability variable of the rightlane is large, and the coefficients b, d, and e of the right lane arefixed as constants b^(v), d^(v), e^(v) with the right lane as thereference lane. Using the least-squares method, the straight line andcurve of the left lane for the path are set by obtaining the lanecoefficient a_(L) ^(v), c_(L) ^(v) for the path passing through theskeleton line coordinates of the left lane.

In the case of step S1305, the lane probability variable of the leftlane is large, and coefficients b, d, and e of the left lane are fixedas constant b^(v), d^(v), e^(v) using the left lane as the referencelane. Using the least-squares method, the straight line and curve of thedriving lane for the path are set by obtaining the lane coefficienta_(R) ^(v), c_(R) ^(v) for the path that passes through the coordinatesof the skeleton line of the right lane.

The lane information for the path of the extended lane uses the laneinformation of the driving lane for the path generated above. Byinputting the coefficient of the left driving lane and the coordinatesof the skeleton line, the coefficient b_(L) ^(v), d_(L) ^(v), e_(L) ^(v)of the left extension lane may be calculated using the least squaresmethod. The remaining coefficient b_(L) ^(x), d_(L) ^(x), e_(L) ^(x) ofleft extended lane are equal to the coefficient b_(L) ^(v), d_(L) ^(v),e_(L) ^(v) of left driving lane. By inputting the coefficient b_(R)^(x), d_(R) ^(x), e_(R) ^(x) of the right driving lane and thecoordinates of the skeleton line, the coefficient a_(R) ^(x), c_(R) ^(x)of the right extension lane can be calculated using the least squaresmethod. The remaining coefficient b_(R) ^(x), d_(R) ^(x), e_(R) ^(x) ofright extended lane are equal to the coefficient b_(R) ^(v), d_(R) ^(v),e_(R) ^(v) of right driving lane.

(Interlocking Method of Multi-Lane and Multi-Path when Changing Lanes)

Hereinafter, when a lane change occurs, a method of interlockingmulti-lanes and multi-paths will be described.

FIGS. 14 to 17 are views continuously illustrating a state in which asubject vehicle changes a location from an initial lane to a left laneaccording to an embodiment of the present invention.

Referring to FIG. 14 , the subject vehicle 10 is in a state beforechanging a lane.

Referring to FIG. 15 , the subject vehicle 10 is in a state just beforechanging the lane to the left, the multi-lane connection structure andthe multi-path connection structure are the same as in FIG. 14 . Thatis, the multi-lane ({circle around (0)}{circle around (1)}{circle around(2)}{circle around (3)}) is connected with the index of the multi-path([4][5][6][7]).

Referring to FIG. 16 , the subject vehicle 10 is immediately after alane change occurs, and at this time the multi-lane moves one step tothe left, and the multi-lane ({circle around (0)}{circle around(1)}{circle around (2)}{circle around (3)}) is connected with the indexof the multi-path ([3][4][5][6]).

Referring to FIG. 17 , the subject vehicle 10 is in a state afterchanging a lane.

A method of detecting a lane change is as follows.

For example, in FIGS. 14 to 17 , a moment when the latest x-coordinatevalue of the path [5] changes from a negative number to a positivenumber is used as a lane change condition. This phenomenon occursbecause the coordinate system used in the present invention uses thecamera coordinate system (orthogonal arrow in the drawing) of thesubject vehicle. In FIG. 15 , the x coordinate value of the path [5] isnegative, and in FIG. 16 , the x coordinate value of the path [5] ispositive.

Summarizing the above in a formula, it is as follows:

(X _(k−1) ^(L)<0)∩(X _(k) ^(L)>0)  [Equation 12]

When the condition of Equation 12 is satisfied, it may be determinedthat the lane has changed to the left. Here, X_(k−1) ^(L) is the X valueof the left lane of the previous frame, and X_(k) ^(L) is the X value ofthe left lane of the current frame.

In the world coordinate system, (X, Y)=(0, 0) is the position of thecamera installed on the subject vehicle. In other words, if the leftlane is located to the left in the previous frame and to the right inthe current frame of the reference value, it may be determined that thelane is changed to the left.

The following Equation 13 is a discriminant for occurrence of a lanechange in the subject vehicle to the right.

(X _(k−1) ^(R)<0)∩(X _(k) ^(R)>0)  [Equation 13]

Here, X_(k−1) ^(R) is the X value of the right lane of the previousframe, and X_(k) ^(R) is the X value of the right lane of the currentframe.

When the lane change of the subject vehicle occurs, the location of themulti-lane changes with respect to the path as shown in FIGS. 14 to 17 .In this case, information on the multi-lane must be moved. Multi-laneinformation typically includes coefficients (a, b, c, d, e in Equations10 and 11) of straight-line and curved equations.

FIGS. 18 and 19 are diagrams illustrating a method of moving multi-laneinformation when a subject vehicle changes lanes.

Referring to FIG. 18 , when a subject vehicle changes into a left lane,a method of moving multi-lane information is shown. The circular numbermeans the circular number of FIGS. 14 and 15 . FIG. 18 is an operationperformed immediately after changing into a left lane.

When explaining the meaning of each row in FIG. 18 , information on theright extended lane {circle around (3)} is deleted. Copy the informationon the right driving lane {circle around (2)} to the right extensionlane {circle around (3)}. Copy the information from the left drivinglane {circle around (1)} to the right driving lane {circle around (2)}.Copy the information on the left extension lane {circle around (0)} tothe left driving lane {circle around (1)}. Copy the new lane informationto the left extended lane {circle around (0)}.

Referring to FIG. 19 , when a subject vehicle changes into a right lane,a method of moving multi-lane information is shown.

When explaining the meaning of each row in FIG. 19 , information on theleft extended lane {circle around (0)} is deleted. Copy the informationon the left driving lane {circle around (1)} to the left extension lane{circle around (0)}. Copy the information from the right driving lane{circle around (2)} to the left driving lane {circle around (1)}. Copythe information from the right extension lane {circle around (3)} to theright driving lane {circle around (2)}. Copy the new lane information tothe right extension lane {circle around (3)}.

(Generate Path of Subject Vehicle)

Hereinafter, a method for generating a path of a subject vehicle will bedescribed. Since the path of the subject vehicle can be estimated usingthe path of the driving lane, it can be obtained using the method ofcreating one path of FIG. 8 .

The average value of the coefficients of the equation of the drivinglane is used as input information in step S801 of FIG. 8 .

Equation 14 shows a method of calculating path input information of asubject vehicle.

$\begin{matrix}{a_{S}^{v} = 0} & \left\lbrack {{Equation}14} \right\rbrack\end{matrix}$ $b_{S}^{v} = \frac{b_{L}^{v} + b_{R}^{v}}{2}$c_(S)^(v) = 0 $d_{S}^{v} = \frac{d_{L}^{v} + d_{R}^{v}}{2}$$e_{S}^{v} = \frac{e_{L}^{v} + e_{R}^{v}}{2}$

In Equation 14, a_(S) ^(v) and b_(S) ^(v) denote a coefficient of theequation of a straight line (Equation 8), c_(S) ^(v), d_(S) ^(v), ande_(S) ^(v) denote a coefficient of the equation of a curve (Equation11). b_(S) ^(v), d_(S) ^(v), and e_(S) ^(v) denote the slope andcurvature of the path of the subject vehicle. The average value of theslope and curvature of the driving lane may be estimated as acoefficient of the equation of the path of the subject vehicle. a_(S)^(v) and c_(S) ^(v) mean the position of the x-axis of the subjectvehicle, and since the coordinate system based on the present inventionis the world coordinate system with the position of the camera of thesubject vehicle as the origin, a_(S) ^(v) and c_(S) ^(v) have a value of0.

If the subject vehicle information obtained in Equation 14 is input instep S801 of FIG. 8 , a path set of the subject vehicle may be obtainedin step S805.

FIG. 20 is a diagram illustrating a path of a subject vehicle accordingto an embodiment of the present invention.

Referring to FIG. 20 , the path of the subject vehicle 10 obtained byusing the above-described method is shown, and arrows displayed at thelower end of the subject vehicle mean the path of the subject vehicle.

(How to Create a Path for the Front Vehicle)

Hereinafter, a method for generating a path for front vehicle will bedescribed.

Front vehicle seen by the front camera of the subject vehicle can berecognized using a deep learning method using tensorflow. Worldcoordinate values can be obtained with the information on the positionsof the front vehicle, but these are coordinate values with the subjectvehicle as the origin of the coordinate axis.

FIGS. 21 to 23 are diagrams for explaining a method for generating apath of a front vehicle using a motion vector of a subject vehicleaccording to an embodiment of the present invention. In order to findthe path of the front vehicle, we show how to find the motion vector vof the front vehicle in the world coordinate system.

Referring to FIG. 21 , when the camera coordinates of the subjectvehicle are taken as the origin, the vector b means that the frontvehicle moves from a specific position of the image of the previousframe to a specific position of the image of the current frame as avector. It is the movement of the front vehicle that can be obtainedonly from images captured by the camera without input of otherinformation. In other words, it has the same meaning as when there is nomovement of the subject vehicle. The starting point b_(S) of the vectorb means the coordinates of the front vehicle of the previous frame, andthe end point b_(E) means the coordinates of the front vehicle of thecurrent frame.

In FIG. 21 , a vector h denotes a motion vector of a subject vehicle.The starting point h_(S) of the vector ii means the coordinates of thesubject vehicle of the previous frame, and the end point h_(E) means thecoordinates of the subject vehicle of the current frame. The motionvector of the subject vehicle refers to a connection between the currentcoordinates and the coordinates of the previous frame in the path of thesubject vehicle. A method of obtaining the path of the subject vehiclehas been previously described.

Referring to FIG. 22 , a state of finding a vector addition v of avector b and a vector h is shown. In the world coordinate system, themotion vector v of the front vehicle means that the motion vector h ofthe subject vehicle is added to the vector b.

In the case of FIG. 22 , the starting point of the vector addition v andthe starting point of the vector b are the same, but the ending pointsb_(E) and v_(E) exist at different positions. Since we want to maintainthe position of the end point b_(E) (meaning the current position of thefront vehicle) of the existing vector b, we use a different version ofthe vector addition shown in FIG. 23 .

Referring to FIG. 23 , using this method, the vector v maintains thecurrent position v_(E) of the front vehicle, and it is possible to knowthe position v_(S) of the front vehicle of one frame before.

FIG. 24 is a top view showing a motion vector of a front vehicleaccording to an embodiment of the present invention.

Referring to FIG. 24 , a vehicle denotes the subject vehicle 10, and amotion vector h of the subject vehicle denotes a bottom arrow. Thevehicle means the front vehicle 20, and the motion vector v of the frontvehicle means the upper arrow.

In the embodiment, the subject vehicle and the front vehicle are movingat a constant velocity at a speed of about 70 km/h, and since the vectorb has a relatively small vector quantity, the vector component of thesubject vehicle is successfully applied to the vector calculation of thefront vehicle. Additionally, the motion vector v denotes theinstantaneous speed of the front vehicle.

The vector v of the front vehicle described above is for the currentframe. The vectors v_(k) obtained in the past time are stored in thememory in the form of a set.

Hereinafter, a method of concatenating vector sets into a path set willbe described.

FIG. 25 is a view illustrating a general state in which vectors of afront vehicle are connected to each other to form a path set accordingto an embodiment of the present invention.

Referring to FIG. 25 , since the start point of the previous vector andthe end point of the current vector are always connected in the pathset, coordinates are defined as

{(x ₀ ,y ₀),(x ₁ ,y ₁), . . . ,(x _(n) ,y _(n))}.

The vector component of v₁ is (a₁, b₁) and the vector component of v_(n)is (a_(n), b_(n)).

In FIG. 25 , the coordinates of the end point v_(cE) of v₂ are the sameas the coordinates of the start point v_(1S) of v₁. If the coordinatesof the starting point v_(2s) of v₂ is (x₂, y₂), the coordinates of (x₂,y₂) can be obtained as in Equation 15 by subtracting the vectorcomponent (a₂, b₂) from the starting point (x₁, y₁) of v₁.

x ₂ =x ₁ −a ₂

y ₂ =y ₁ −b ₂  [Equation 15]

If Equation 15 is expanded into a general expression that can becalculated for all path coordinates, Equation 16 is obtained.

x _(k+1) =x _(k) −a _(k+1)

y _(k+1) =y _(k) −b _(k+1)

1≤k≤(n−1)  [Equation 16]

In Equation 16, the range of k is limited to 1 to n−1. n means thenumber of total vectors. Here, the starting point (x₁, y₁) and theending point (x₀, y₀) of the vector v₁ should be input as initialvalues. The starting point (x₁, y₁) and the ending point (x₀, y₀) meanv_(S) and v_(E) in FIG. 23 , and the method of obtaining the coordinateshas been described above.

All path coordinates of the front vehicle can be obtained using theabove method, and red arrows in FIG. 24 show an embodiment in which pathcoordinates of the front vehicle (red) are obtained.

While the present invention has been particularly shown and describedwith reference to preferred embodiments thereof, it will be understoodby those of ordinary skill in the art that various changes in form anddetails may be made therein without departing from the spirit and scopeof the present invention as defined by the following claims

What is claimed is:
 1. Vehicle path restoration system throughsequential image analysis, comprising: an image capturing unit thatacquires sequential images from the front camera installed in thesubject vehicle; an image analysis unit for generating multiple lanesthat can be recognized from the sequential images of the video fileacquired by the image capturing unit and multi-paths calculated usingthe geometric characteristics of the lanes recognized at the currenttime and the speed of the subject vehicle, that restores the path of thesubject vehicle and restores the path of the front vehicle driving infront of the subject vehicle; a memory for storing path data of thesubject vehicle and the front vehicle restored by the image analysisunit; and a display unit that expresses the path data of the subjectvehicle and the front vehicle stored in the memory in the form of a topview.
 2. The vehicle path restoration system through sequential imageanalysis of claim 1, wherein the multi paths are located by a pluralityof indexes in a predefined memory space, so that it is possible to storepath data even when the subject vehicle changes lanes left and rightbetween the multi paths.
 3. Vehicle path restoration method throughsequential image analysis, comprising: acquiring sequential images froma front camera of an image capturing unit installed in a subjectvehicle; generating a path of a subject vehicle and a front vehicle byreceiving the sequential image and performing image analysis in an imageanalysis unit; and expressing the image analysis of the image analysisunit in the form of a top view on the display unit, wherein performingimage analysis in the image analysis unit comprises: determining a laneof the current time; generating a multi-lane recognizable in the frontcamera image and a multi-path for each index calculated using geometriccharacteristics of the lane recognized at the current time and the speedof the subject vehicle; generating path data of the subject vehicle; andgenerating path data of the front vehicle.
 4. The vehicle pathrestoration method through sequential image analysis of claim 3, whereincreating a path set of each index in the multi-path comprises: receivinginput information that is input data for creating a path set;determining whether the path is a curved line or a straight lane;generating a curved path when the path is determined as a curved lane,and generating a straight path when the path is determined as a straightlane; and creating a path set of the corresponding index by integratingthe curved path and the straight path.
 5. The vehicle path restorationmethod through sequential image analysis of claim 4, wherein the inputdata for creating the path set are the speed of the subject vehicle, thepath index number, the equation coefficient of a straight line and theequation coefficient of the curve suitable for the lane recognized inthe image, the radius of curvature, and the curve discriminationcoefficient.
 6. The vehicle path restoration method through sequentialimage analysis of claim 3, wherein generating the multi-path for eachindex includes: determining an index of a path connected to the lanerecognized in the current frame in the front camera image as an activepath index, and determining an index of a path not connected to the lanerecognized in the current frame in the front camera image as an inactivepath index; the active path index is passed to the active multi-path togenerate an active multi-path; and the inactive path index istransferred to the inactive multi-path, and in the inactive multi-pathgeneration, a new path is not added, but according to the movement ofthe subject vehicle, correcting the location of the path stored in thepast time.
 7. The vehicle path restoration method through sequentialimage analysis of claim 6, wherein the active multi-path generation isthe entire process of a method of creating a path set of each individualindex, and the active single path generation step is performed severaltimes as many as the number of lanes recognized in the current image. 8.The vehicle path restoration method through sequential image analysis ofclaim 3, wherein generating path data of the subject vehicle comprises:obtaining equation coefficients of straight lines and the equationcoefficients of curve of a driving lane, which are both lanes of asubject vehicle; determining the equation coefficients of the straightline and the equation coefficients of curve for the path of the subjectvehicle by using the equation coefficients of the straight line and theequation coefficients of the curve of the driving lane; determiningwhether the path is a curved line or a straight lane; generating acurved path when the path is determined as a curved lane, and generatinga straight path when the path is determined as a straight lane; andgenerating a set of paths for the subject vehicle by integrating thecurved path and the straight path.
 9. The vehicle path restorationmethod through sequential image analysis of claim 8, further comprising:making the driving lane in a parallel state: determining a laneprobability variable that is a probability of being a lane with respectto a right lane and a left lane; when the difference between the leftlane and the right lane probability variable is less than or equal to apredetermined error tolerance, obtaining a median value of the slope andcurvature of the two lane information, and calculating the equations ofstraight lines and curves for the path with respect to the left andright lanes; comparing the sizes of the left lane and right laneprobability variables when the difference between the left and rightlane probability variables is out of a predetermined error tolerance; ifthe right-lane probability variable is larger than the left-laneprobability variable, calculate the equations of straight lines andcurves for the left-lane path, such as the slope and curvature of theright-lane; and if the right-lane probability variable is smaller thanthe left-lane probability variable, calculate the equations of straightlines and curves for the right-lane path, such as the slope andcurvature of the left-lane.
 10. The vehicle path restoration methodthrough sequential image analysis of claim 3, wherein when the locationof the multi-lane is changed because the subject vehicle is changed to alane, the information of the multi-lane is moved together.
 11. Thevehicle path restoration method through sequential image analysis ofclaim 3, wherein generating of the path data of the front vehicleincludes: determining a vector b representing that the front vehiclemoves from a specific position of an image of a previous frame to aspecific position of an image of a current frame when the cameracoordinates of the subject vehicle are taken as the origin; determininga motion vector h of the subject vehicle connecting the currentcoordinates and the coordinates of the previous frame in the path of thesubject vehicle; and obtaining a motion vector v of the front vehiclethat is a vector addition of the vector b and the vector h.